Papers with calibration of quantized models
When Quantization Affects Confidence of Large Language Models? (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing studies have shown that quantization compromises performance and exacerbates biases in Large Language Models. |
| Approach: | They propose an explanation for quantization loss based on confidence levels . they propose a range of efficient compression and acceleration methods including quan-tization . |
| Outcome: | The proposed methods show that quantization decreases confidence regarding true labels and that it exacerbates biases across different scales. |